Dismantling Of The Whole Process Of Shopee Taiwan Store Group Operation From Product Selection To Shelf Launch And Implementation Suggestions

1.

overview and definition of technical goals

- goal: build a multi-store (store group) operation support platform on shopee taiwan to ensure stable and low-latency crawling, analysis, shelving, and image/resource distribution.
- key technical points: vps/host selection, domain name and dns strategy, cdn and static resource distribution, ddos protection and load balancing.
- performance indicators: ttfb < 200ms, first screen of the page ≤ 1.2s, stable load concurrency of 200-500 requests/second.
- quantifiable goals: 50,000 items to be captured per day; automatic listing success rate > 98%; estimated bandwidth of 3tb/month.
- risk points: platform blocked ip addresses, ddos attacks, database bottlenecks, image storage delays, all of which need to be prevented and mitigated at the architectural level.

2.

server deployment of product selection and data capture system

- it is recommended that the crawling task be deployed in a decentralized manner: each crawling node uses an independent vps to avoid single-point ip being blocked.
- suggested node configuration examples: 4 vcpu / 8gb ram / 80gb nvme / 2tb bandwidth, os: ubuntu 20.04.
- tools and services: use scrapy/playwright for dynamic crawling, combined with redis task queue and postgresql storage.
- task scheduling: graded crawling every minute/hour, short polling for key products, and long polling for unpopular products to reduce load.
- logging and monitoring: prometheus + grafana monitors cpu, io, network and error rate, and the alert is set to trigger when the error rate is 5%.

3.

vps/host selection, server software and performance tuning

- mainstream supplier selection: aws lightsail, digitalocean, vultr, gcp (taiwan) or local taiwanese computer room suppliers.
- recommended architecture: front-end nginx reverse proxy + php/python application service + mariadb/mysql master-slave + redis cache.
- specific software version examples: nginx 1.18, php-fpm 7.4 or python 3.9, mysql 8.0, redis 6.0.
- focus on performance tuning: adjust nginx worker_processes and worker_connections, enable keepalive and gzip, and set innodb_buffer_pool_size=4g for mysql (taking an 8gb ram machine as an example).
- disk and network: it is recommended to use nvme ssd and monitor i/o wait. if it exceeds 20%, consider upgrading the disk or adopting a sharding solution.

4.

domain name, dns and cdn configuration details

- domain name management: the main domain name is separated from the subdomain name, images/static resources use cdn.example.com, and the backend api uses api.example.com.
- dns selection: use cloudflare dns or other high-availability dns, ttl settings: low ttl for ever-changing resources, high ttl for static resources.
- cdn strategy: static images and js/css use cdn to place taiwan’s main pops in taipei/taichung to reduce delays.
- ssl & security: auto-renew with let's encrypt or endpoint ssl by cloudflare, and enforce https.
- load and cache rules: static content is cached for 7 days, dynamic api is set to short cache or no cache, and details can be adjusted with cache-control.

5.

shelf automation platform and api design

- dismantling of the listing process: data import → image processing and cdn upload → api call to shopee for listing → status writeback and retry.
- image processing suggestions: use a dedicated image bed service or s3 compatible storage. the images are cropped and compressed on the back end before being pushed to cdn.
- api call concurrency control: each listed node limits the number of simultaneous connections to 10-20 to avoid being restricted by marketplace.
- retry mechanism: adopt exponential backoff, retry up to 5 times, and record each error code.
- logs and auditing: 90-day logs of all listing requests are kept for traceback of exceptions or appeals.

6.

implementation of ddos protection and high availability architecture

- border protection: use cloudflare or similar waf/cdn for edge protection, and block common attacks and bot by default.
- network layer protection: if heavy traffic is expected, use anycast + elastic bandwidth and sign a ddos sla with the computer room.
- architecture redundancy: multiple front-end nodes are scattered in different computer rooms, and the back-end database adopts master-slave or cluster, and is backed up regularly.
- auto-scaling: use automated scripts or containerization (docker/kubernetes) to spin up more nodes during traffic peaks.
- drills and monitoring: conduct ddos drills every quarter, and set 1-minute level traffic alarms and automated blocking strategies.

7.

real cases and server configuration data examples (including data tables)

- case background: a taiwanese e-commerce team operates 12 shopee stores and established an internal listing platform, image bed and crawling node.
- initial configuration: 3 4vcpu/8gb vps as crawling nodes, 2 4vcpu/16gb vps as application servers, and 1 8vcpu/32gb as main database and backup node.
- optimization results: after the automation of the shelves, the daily shelf volume increased from 800 to 3,200; the average page loading time dropped from 3.2s to 1.1s; the bandwidth usage dropped from 4tb/month to 2.6tb/month (due to cdn improving cache).
- ddos actual combat: a 150gbps attack was mitigated by cloudflare, and the backend server cpu was maintained at an average of 15%-30%.
- the following table shows the three groups of node configurations and measured indicators:

node	configuration	average cpu	average ttfb	concurrency per second
grab nodes	4 vcpu/8gb/80gb nvme/2tb	25%	350ms	50-100
application server	4 vcpu/16gb/160gb nvme/4tb	30%	180ms	200-400
database master node	8 vcpu/32gb/500gb nvme/10tb	40%	120ms	500 (query)

- recommendation: based on the above data, use at least 2 application nodes + 3 crawling nodes as the starting point, closely integrate with cdn and waf, and regularly adjust resources based on monitoring indicators.